Multiple Time Steps Research Articles

Multiple timestep reversible N-body integrators for close encounters in planetary systems

Abstract We present new “almost” time-reversible integrators for solution of planetary systems consisting of “planets” and a dominant mass (“star”). The algorithms can be considered adaptive generalizations of the Wisdom–Holman method, in which all pairs of planets can be assigned timesteps. These timesteps, along with the global timestep, can be adapted time-reversibly, often at no appreciable additional compute cost, without sacrificing any of the long-term error benefits of the Wisdom–Holman method. The method can also be considered a simpler and more flexible version of the SYMBA symplectic code. We perform tests on several challenging problems with close encounters and find the reversible algorithms are up to 2.6 times faster than a code based on SYMBA. The codes presented here are available on Github. We also find adapting a global timestep reversibly and discretely must be done in block-synchronized manner or similar.

Conformal Prediction Regions for Time Series Using Linear Complementarity Programming

Conformal prediction is a statistical tool for producing prediction regions of machine learning models that are valid with high probability. However, applying conformal prediction to time series data leads to conservative prediction regions. In fact, to obtain prediction regions over T time steps with confidence 1--delta, previous works require that each individual prediction region is valid with confidence 1--delta/T. We propose an optimization-based method for reducing this conservatism to enable long horizon planning and verification when using learning-enabled time series predictors. Instead of considering prediction errors individually at each time step, we consider a parameterized prediction error over multiple time steps. By optimizing the parameters over an additional dataset, we find prediction regions that are not conservative. We show that this problem can be cast as a mixed integer linear complementarity program (MILCP), which we then relax into a linear complementarity program (LCP). Additionally, we prove that the relaxed LP has the same optimal cost as the original MILCP. Finally, we demonstrate the efficacy of our method on case studies using pedestrian trajectory predictors and F16 fighter jet altitude predictors.

Breaking Time Invariance: Assorted-Time Normalization for RNNs

Methods such as Layer Normalization (LN) and Batch Normalization have proven to be effective in improving the training of Recurrent Neural Networks (RNNs). However, existing methods normalize using only the instantaneous information at one particular time step, and the result of the normalization is a preactivation state with a time-independent distribution. This implementation fails to account for certain temporal differences inherent in the inputs and the architecture of RNNs. Since these networks share weights across time steps, it may also be desirable to account for the connections between time steps in the normalization scheme. In this paper, we propose a normalization method called Assorted-Time Normalization (ATN), which preserves information from multiple consecutive time steps and normalizes using them. This setup allows us to introduce longer time dependencies into the traditional normalization methods without introducing any new trainable parameters. We present theoretical derivations for the gradient propagation and prove the weight scaling invariance property. Our experiments applying ATN to LN demonstrate consistent improvement on various tasks, such as Adding, Copying, and Denoise Problems and Language Modeling Problems.

Microgrid Formation and Real-Time Scheduling of Active Distribution Networks Considering Source-Load Stochasticity

The post-disruption microgrid (MG) formation and the subsequent scheduling are resilience-enhancing measures for active distribution networks (ADNs) against disastrous events. This paper proposes an integrated MG formation and scheduling solution, considering stochastic loads and distributed generators (DGs). Specifically, a first-stage MG formation model and a sec-ond-stage MG scheduling model are proposed to flexibly identify MG boundaries and restore as many critical loads as possible. The MG formation problem is formulated over multiple time steps to address time-varying generation outputs of intermittent DGs. Moreover, in the MG scheduling problem, the stochasticity of load demands and DG outputs is modeled as joint chance constraints (JCCs) to describe the possibility of security violations in the over-all system. An efficient approximation method is next developed to convexify the JCCs into solvable second-order cones. The interac-tion between the MG formation and scheduling by the model predictive control implementation achieves a responsive solution for enhancing the emergency support capability of ADNs by incor-porating power forecasts and real-time scheduling. The performance of the proposed method is verified in a modified real-world 136-node test feeder under catastrophic failures.

Euler State Networks: Non-dissipative Reservoir Computing

Inspired by the numerical solution of ordinary differential equations, in this paper, we propose a novel Reservoir Computing (RC) model, called the Euler State Network (EuSN). The presented approach makes use of forward Euler discretization and antisymmetric recurrent matrices to design reservoir dynamics that are both stable and non-dissipative by construction.Our mathematical analysis shows that the resulting model is biased towards a unitary effective spectral radius and zero local Lyapunov exponents, intrinsically operating near the edge of stability. Experiments on long-term memory tasks show the clear superiority of the proposed approach over standard RC models in problems requiring effective propagation of input information over multiple time steps. Furthermore, results on time-series classification benchmarks indicate that EuSN can match (or even exceed) the accuracy of trainable Recurrent Neural Networks, while retaining the training efficiency of the RC family, resulting in up to ≈464-fold savings in computation time and ≈1750-fold savings in energy consumption. At the same time, our results on time-series modeling tasks show competitive results against standard RC when the architecture is complemented by direct input-readout connections.

Intelligent fault diagnosis of storage stacking machinery under variable working conditions using attention-based adaptive multimodal feature fusion networks

Due to the harsh working environment of storage stacking machinery, the fault information of important components is significantly complex, which leads to the problem of low classification accuracy and high computational complexity of existing deep learning-based fault diagnosis methods. To alleviate the problem, this paper presents a novel architecture named attention-based adaptive multimodal feature fusion networks for intelligent fault diagnosis of storage stacking machinery, which is aimed at improving the diagnostic precision and robustness of feature fusion network and learning the broader feature representation. Firstly, the long short-term memory layer is introduced to extract the feature information of multiple time steps to improve the self-extraction ability of multi-temporal features. Then, the maximum temporal feature fusion module is utilized to highlight the recognizability of deep fusion features. Finally, a residual layer with spanning connections is added to increase the utilization and characterization capability of deep fusion features. Experimental results demonstrate the effectiveness and superiority of the proposed method in intelligent fault diagnosis of storage stacking machinery under variable working conditions compared with some state-of-the-art deep learning-based methodologies.

Digital twin-driven graph domain adaptation neural network for remaining useful life prediction of rolling bearing

Remaining useful life (RUL) prediction is significant for the healthy operation of machinery. In order to accurately identify the bearing degeneration states, it is necessary to collect massive full lifecycle data. However, the bearing lifecycle data is insufficient for effectively training a RUL prediction model in engineering practice. In this paper, a digital twin-driven graph domain adaptation method is proposed. First, a full lifecycle dynamic twin model of bearings is constructed to generate abundant twin data, in which the surface morphology evolution and roller relative slip at different stages are simulated to generate vibration responses. Second, a novel multi-layered cross-domain gated graph convolutional network (MGGCN) is developed, in which a new graph domain adaptation model is designed to solve the problem that traditional domain adaptation methods are not effective in processing the non-Euclidean data. The spatial and temporal features are extracted by multiple nonlinear transformations and previous time-step hidden state incorporation, respectively. In addition, a graph Laplacian regularized maximum mean discrepancy (GLMMD) is designed and applied in the training of model to enhance the capability of discerning graph domain differences. The experimental results confirm that the proposed method can achieve effective performance even in scenarios with limited actual data.

A novel extreme adaptive GRU for multivariate time series forecasting

Multivariate time series forecasting is a critical problem in many real-world scenarios. Recent advances in deep learning have significantly enhanced the ability to tackle such problems. However, a primary challenge in time series forecasting comes from the imbalanced time series data that include extreme events. Despite being a small fraction of the data instances, extreme events can have a negative impact on forecasting as they deviate from the majority. However, many recent time series forecasting methods neglect this issue, leading to suboptimal performance. To address these challenges, we introduce a novel model, the Extreme Event Adaptive Gated Recurrent Unit (eGRU), tailored explicitly for forecasting tasks. The eGRU is designed to effectively learn both normal and extreme event patterns within time series data. Furthermore, we introduce a time series data segmentation technique that divides the input sequence into segments, each comprising multiple time steps. This segmentation empowers the eGRU to capture data patterns at different time step resolutions while simultaneously reducing the overall input length. We conducted comprehensive experiments on four real-world benchmark datasets to evaluate the eGRU’s performance. Our results showcase its superiority over vanilla RNNs, LSTMs, GRUs, and other state-of-the-art RNN variants in multivariate time series forecasting. Additionally, we conducted ablation studies to demonstrate the consistently superior performance of eGRU in generating accurate forecasts while incorporating a diverse range of labeling results.

Integrated surrogate framework for reactive transport simulation of uranium in situ leaching with generative models

The heterogeneity of uranium-bearing aquifers plays a crucial role in governing the migration and leaching of uranium. Uncertainty quantification (UQ) based on the Monte Carlo method is usually used to deal with the issue of aquifer heterogeneity. However, the substantial computational costs of this method arise from a large number of calls to the uranium reactive transport simulation; therefore, surrogate models are developed to reduce these costs. This study leverages a conditional deep convolutional generative adversarial network (cDC-GAN) and an adversarial autoencoder-based kriging regression (AAE-KRG) to develop an integrated surrogate framework for uranium reactive transport simulation. cDC-GAN and AAE-KRG are designed to achieve the cross-domain mapping of the permeability coefficient field to the uranium spatial distribution and leaching concentration over time, respectively. The results demonstrate a structural similarity exceeding 0.9 between the cDC-GAN and the numerical simulation, with a mean square error generally less than 0.002. These findings suggest that the cDC-GAN exhibits high accuracy and stability in terms of replacing the numerical simulation to obtain the spatial-temporal evolution of the uranium plume. Additionally, cDC-GAN is a dynamic surrogate model with the capability of interpolation because uranium spatial distributions with multiple time steps are used as training data during the training process. AAE-KRG also exhibits high accuracy and stability in terms of predicting the leaching concentration by permeability coefficient fields, as evidenced by determination coefficients generally exceeding 0.95. By comparing the computational costs, it is clear that cDC-GAN and AAE-KRG are suitable for processing spatial distribution and time series data, respectively. The trained surrogate models are used to undertake UQ of the uranium leaching area and leaching concentration. The results indicate that the uncertainty of the leaching area in the field-test model increases with time, and the increasing rate reaches a maximum in the middle period (201–300 d). The uncertainty of the leaching concentration increases with time in the early period, decreases after peaking at 137 d, and stabilizes in the middle and late periods. It is observed that the change characteristics of the uncertainty are consistent with the values of the leaching area or concentration. Furthermore, the spatial distribution pattern of uranium pollution is determined based on stochastic simulation results. The uncertainty intervals and the pollution risk status at multiple time steps are quickly obtained by surrogate models, which is helpful for scheme optimization in uranium mining. Owing to its characteristics of being data-driven and highly adaptable to nonlinear data, the integrated surrogate framework has applications beyond uranium in situ leaching.

Soil Moisture Prediction Model Based on Improved GRU Recurrent Neural Network

Soil moisture plays a crucial role in land water and energy cycles, and has a certain impact on weather and climate change. In agricultural production, crop moisture status can be determined based on soil moisture, and timely and effective irrigation strategies can be formulated to ensure grain yield while saving water resources, maximizing the value of agricultural water resource utilization, and achieving sustainable development. Therefore, the accuracy of soil moisture prediction has important research value for many fields such as agriculture and climate. In this paper, the super parameters of GRU Recurrent neural network are optimized by intelligent seagull optimization algorithm using a small number of influencing factors, namely, atmospheric temperature, atmospheric humidity, rainfall and soil moisture data, and a soil moisture prediction model is established. The model was used to predict soil moisture for the next 12 hours, 24 hours, 36 hours, and 48 hours, respectively. The final experiment showed that the model in this paper had better predictive effect on soil moisture, with the best predictive evaluation index data being MAPE (12h) = 4.4120%, R2 (12h) = 0.94605, and RMSE (12h) = 1.9998; By comparing the prediction results of multiple time steps vertically, it was found that the prediction accuracy of the model in this paper decreased more smoothly, meeting the requirements of soil moisture prediction.

LEISN: A long explicit–implicit spatio-temporal network for traffic flow forecasting

Recent studies have shown that it is necessary to further improve the prediction accuracy of complex and dynamic traffic flow from two angles: temporal dependence and spatial dependence. Although many spatio-temporal networks are extracting features from both temporal and spatial dependencies, there are still two problems that need to be addressed. Firstly, existing methods often focus on modeling local spatial dependence relationship between adjacent nodes, but neglect non-local spatial dependence relationship between distant nodes. Secondly, while recent work has explored various methods such as convolutional and recurrent neural networks for modeling temporal dependence, they may not fully capture the long-term temporal dependence in traffic data. Therefore, addressing the challenges of long-term dependency, explicit spatial dependency, and implicit spatial dependency, this paper proposes a Long-term Explicit–Implicit Spatio-Temporal Network (LEISN) for traffic flow prediction. A Long-term Dependency Module has been designed to store hidden states generated from multiple previous time steps, facilitating the transmission of long-term features. Based on this, two graph convolution-based spatial feature extraction branches are designed to extract explicit spatial features based on the adjacency matrix generated by spatial topology and implicit spatial features based on the implicit adjacency matrix generated by trend similarity, respectively. Then all spatio-temporal features are fused to produce the next state. Additionally, a new encoder framework, LENSI-ED, was proposed based on LEISN. Comparative experiments are conducted on four datasets, and the results showed that our model has advantages over existing methods. The proposed model addresses the issues of local and non-local spatial dependence relationships and long-term temporal dependence in traffic flow forecasting.

Spatio-Temporal Joint Graph Convolutional Networks for Traffic Forecasting

Recent studies have shifted their focus towards formulating traffic forecasting as a spatio-temporal graph modeling problem. Typically, they constructed a static spatial graph at each time step and then connected each node with itself between adjacent time steps to create a spatio-temporal graph. However, this approach failed to explicitly reflect the correlations between different nodes at different time steps, thus limiting the learning capability of graph neural networks. Additionally, those models overlooked the dynamic spatio-temporal correlations among nodes by using the same adjacency matrix across different time steps. To address these limitations, we propose a novel approach called Spatio-Temporal Joint Graph Convolutional Networks (STJGCN) for accurate traffic forecasting on road networks over multiple future time steps. Specifically, our method encompasses the construction of both pre-defined and adaptive spatio-temporal joint graphs (STJGs) between any two time steps, which represent comprehensive and dynamic spatio-temporal correlations. We further introduce dilated causal spatio-temporal joint graph convolution layers on the STJG to capture spatio-temporal dependencies from distinct perspectives with multiple ranges. To aggregate information from different ranges, we propose a multi-range attention mechanism. Finally, we evaluate our approach on five public traffic datasets and experimental results demonstrate that STJGCN is not only computationally efficient but also outperforms 11 state-of-the-art baseline methods.

A hybrid deep learning approach to improve real-time effluent quality prediction in wastewater treatment plant

Wastewater treatment plant (WWTP) operation is usually intricate due to large variations in influent characteristics and nonlinear sewage treatment processes. Effective modeling of WWTP effluent water quality can provide valuable decision-making support to facilitate their operations and management. In this study, we developed a novel hybrid deep learning model by combining the temporal convolutional network (TCN) model with the long short-term memory (LSTM) network model to improve the simulation of hourly total nitrogen (TN) concentration in WWTP effluent. The developed model was tested in a WWTP in Jiangsu Province, China, where the prediction results of the hybrid TCN-LSTM model were compared with those of single deep learning models (TCN and LSTM) and traditional machine learning model (feedforward neural network, FFNN). The hybrid TCN-LSTM model could achieve 33.1 % higher accuracy as compared to the single TCN or LSTM model, and its performance could improve by 63.6 % comparing to the traditional FFNN model. The developed hybrid model also exhibited a higher power prediction of WWTP effluent TN for the next multiple time steps within eight hours, as compared to the standalone TCN, LSTM, and FFNN models. Finally, employing model interpretation approach of Shapley additive explanation to identify the key parameters influencing the behavior of WWTP effluent water quality, it was found that removing variables that did not contribute to the model output could further improve modeling efficiency while optimizing monitoring and management strategies.

Quasi-in-Situ Analysis of Electropolished Additively Manufactured Stainless Steel Surfaces

Progress in additive manufacturing is leading to the emergence of new areas of application. Laser Powder Bed Fusion (L-PBF) is increasingly used for the development of metallic medical implants, but for high-risk implants like vascular support structures (stents), surface quality is critical to ensure successful implantation without harming the surrounding tissue and ensure the patients’ health. Therefore, enhancing the surface quality is crucial. Electropolishing is a method for removing surface roughness by smoothing out micro-peaks and valleys. However, L-PBF structures have a high surface roughness due to metal particles adhering on the surface. To achieve a smooth surface for additively manufactured implants like stents using electropolishing, the removal of these particles needs to be studied in more detail.The objective of this study is to examine the electropolishing mechanism of 316L stainless steel samples additively manufactured through Laser Powder Bed Fusion (L-PBF). The main objective is to investigate the removal properties and surface characteristics during electropolishing. To achieve this, various surfaces were characterized for morphology and roughness during Hull cell experiments. Markings are utilized on the Hull cell sample surfaces to identify points of interest during quasi-in-situ measurements. The surfaces are then analyzed after multiple time steps, applying different currents to investigate particle dissolution. The surface characteristics are analyzed through scanning electron microscopy, and surface roughness is analyzed using laser scanning microscopy.The results show that the electropolishing process preferentially removes the adhering particles present on the surface of the samples. Increasing the current density results in faster particle dissolution and a smoother surface (see Figure 1a and b). The mechanism of material removal of various surface features, as shown in Figure 1 (red circle, yellow arrow and red square), was assessed based on the experimental results of the surface structures seen on the SEM images. It was found that different surface features were removed during the experiment at different polishing times and current densities. The amount of charge flowed was found to correlate with surface morphology.Based on the obtained results, various surface features (such as large adherent particles, agglomerates of smaller particles, and valleys) and their changes with increasing test duration and current density were observed by quasi-in situ analyses. A reduction in the diameter of round particles adhering to the surface was observed at both low and higher current densities (see Figure 1a red circle a). Increasing the polishing time resulted in leveling of both large particles and valleys (see Figure 1b red square). Also, dissolution of agglomerates of smaller particles occurred at different polishing times as a function of current density and polishing time (see Figure 1a yellow arrow) are observed.Smoothed surface structures can be observed in regions with equivalent surface charge density (see Figure 2). As a result, comparable surface morphologies may appear at the same area charge density, irrespective of a specific current density. So, it may be adequate to only consider the amount of charge flowed to describe the electropolishing of additive materials.In conclusion, comprehending the dissolution characteristics of particles on L-PBF surfaces is essential for attaining satisfactory surface finish in electropolishing. The results of this study offer valuable perspectives into the electropolishing mechanism of additively manufactured 316L stainless steel and can guide future investigations on surface finishing and polishing of additive manufactured implants like stents. Figure 1

A Fast Algorithm for Aperiodic Linear Stencil Computation using Fast Fourier Transforms

Stencil computations are widely used to simulate the change of state of physical systems across a multidimensional grid over multiple timesteps. The state-of-the-art techniques in this area fall into three groups: cache-aware tiled looping algorithms, cache-oblivious divide-and-conquer trapezoidal algorithms, and Krylov subspace methods. In this article, we present two efficient parallel algorithms for performing linear stencil computations. Current direct solvers in this domain are computationally inefficient, and Krylov methods require manual labor and mathematical training. We solve these problems for linear stencils by using discrete Fourier transforms preconditioning on a Krylov method to achieve a direct solver that is both fast and general. Indeed, while all currently available algorithms for solving general linear stencils perform Θ( NT ) work, where N is the size of the spatial grid and T is the number of timesteps, our algorithms perform o ( NT ) work. To the best of our knowledge, we give the first algorithms that use fast Fourier transforms to compute final grid data by evolving the initial data for many timesteps at once. Our algorithms handle both periodic and aperiodic boundary conditions and achieve polynomially better performance bounds (i.e., computational complexity and parallel runtime) than all other existing solutions. Initial experimental results show that implementations of our algorithms that evolve grids of roughly 10 7 cells for around 10 5 timesteps run orders of magnitude faster than state-of-the-art implementations for periodic stencil problems, and 1.3× to 8.5× faster for aperiodic stencil problems. Code Repository: https://github.com/TEAlab/FFTStencils

Advances in using mathematical optimization to manage floods with assessment of possible benefits using a case study

Abstract This paper presents the benefits of using mathematical optimization for reservoir operation based on the assumed availability of short-term runoff forecasts. The novelty is the inclusion of the SSARR hydrological routing as optimization constraints in multiple time step optimization, where the routing coefficients are adjusted dynamically as functions of the channel flows. The paper shows significant reduction of downstream peak flows in flood-prone areas even with a forecast horizon of only 2 days, and it also includes the results of testing the effects of different lengths of forecasting horizons on model results. The case study is conducted on the Damodar River Basin in the Indian State of West Bengal, where basin development started in the 1950s, with flood protection of the downstream river valley as the highest management priority, in addition to water supply and hydro power. The solution methodology and the model results presented in this paper pave the way for eventual introduction to automated management of reservoir outflows that could shape water resources industry in much the same way that auto-pilot and driverless cars are revolutionizing the transportation industry, assuming that runoff forecasting capabilities continue to improve.

An incremental singular value decomposition approach for large-scale spatially parallel & distributed but temporally serial data – applied to technical flows

The article presents a strategy and its algorithm to compile a simulation-accompanying, incremental Singular Value Decomposition (SVD) for time-evolving, spatially parallel discrete data sets. The framework addresses state-of-the-art PDE solvers for computational science and engineering applications. An important characteristic of such applications is that the spatial size of the data is often time-invariant and significantly exceeds the temporal size due to the large computational grid in 3D applications. Typical examples, which are also considered in this article, relate to results extracted from unsteady flow simulations. Herein, the flow data, which progresses over time, is frequently calculated spatially parallel based on domain decomposition strategies, which allow to parallelize the simulation on distributed memory machines following a Single Instruction Multiple Data (SIMD) concept.With a view to the memory-efficient reuse of (compressed) simulation results and their CPU time-saving, sufficiently accurate generation, the paper scrutinizes the efficiency of incremental/parallel SVD approaches for such simulation examples. To improve the computational efficiency, the introduction of a bunch matrix is proposed, which enables the aggregation of multiple time steps and SVD updates, and significantly increases the efficiency. The suggested strategy is verified and validated by simple 2D laminar single-phase flows and subsequently applied to more complex 2D and 3D turbulent two-phase flows. Emphasis is given to (a) the accuracy of SVD-based reconstruction, (b) the physical realizability of the reconstructed quantities, (c) the independence of domain partitioning, (d) an efficient snapshot bunching, and (e) related implementation aspects. In addition, the influence of lower and (adaptive) upper rank thresholds on the effort and accuracy is evaluated. A final application renders the practical benefits of the approach and refers to a merchant ship in head waves at Re = 1.4×107 and Fn = 0.26. The simulation involves 2880 processor cores and the related full-rank snapshot matrix has (108×104) entries. With a numerical overhead of O(10%), this snapshot matrix can be incrementally generated and compressed by O(95%). The compression is accompanied by only small errors in the integral force and local wave elevation of O(10−2%). This qualifies the method for an efficient subsequent data processing.

Research on the Prediction of Tire Radial Load Based on 1D CNN and BiGRU

As an important indicator of vehicle systems, tire load is a key factor in the structural design and safety assessment of vehicles. Direct measurement methods for tire loads are expensive and complicated, while conventional load identification methods are limited by low accuracy and poor robustness. This study aimed to propose a radial load identification method for rubber-tired vehicles based on a one-dimensional convolutional neural network (1D CNN) and bidirectional gated recurrent unit (BiGRU). Considering a priori information of the radial load data of tires and based on the observability of the vehicle vibration system, the proposed method selected feature sets and then retained the effective feature subsets through feature selection to construct samples with multiple time steps as input and with a single time step as output for network training. In doing so, the load prediction results were obtained, and the theoretical model was modified by integrating prediction accuracy, generalization performance, and robustness. Compared with traditional algorithms, the proposed method could effectively reduce the error of load identification, improve adaptability under different operating conditions, and handle the measurement error of different noise levels, which are of practical application value in the engineering field.

Stabilizing machine learning prediction of dynamics: Novel noise-inspired regularization tested with reservoir computing

Recent work has shown that machine learning (ML) models can skillfully forecast the dynamics of unknown chaotic systems. Short-term predictions of the state evolution and long-term predictions of the statistical patterns of the dynamics (“climate”) can be produced by employing a feedback loop, whereby the model is trained to predict forward only one time step, then the model output is used as input for multiple time steps. In the absence of mitigating techniques, however, this feedback can result in artificially rapid error growth (“instability”). One established mitigating technique is to add noise to the ML model training input. Based on this technique, we formulate a new penalty term in the loss function for ML models with memory of past inputs that deterministically approximates the effect of many small, independent noise realizations added to the model input during training. We refer to this penalty and the resulting regularization as Linearized Multi-Noise Training (LMNT). We systematically examine the effect of LMNT, input noise, and other established regularization techniques in a case study using reservoir computing, a machine learning method using recurrent neural networks, to predict the spatiotemporal chaotic Kuramoto–Sivashinsky equation. We find that reservoir computers trained with noise or with LMNT produce climate predictions that appear to be indefinitely stable and have a climate very similar to the true system, while the short-term forecasts are substantially more accurate than those trained with other regularization techniques. Finally, we show the deterministic aspect of our LMNT regularization facilitates fast reservoir computer regularization hyperparameter tuning.

SpikeSim: An End-to-End Compute-in-Memory Hardware Evaluation Tool for Benchmarking Spiking Neural Networks

Spiking Neural Networks (SNNs) are an active research domain towards energy efficient machine intelligence. Compared to conventional artificial neural networks (ANNs), SNNs use temporal spike data and bio-plausible neuronal activation functions such as Leaky-Integrate Fire/Integrate Fire (LIF/IF) for data processing. However, SNNs incur significant dot-product operations causing high memory and computation overhead in standard von-Neumann computing platforms. To this end, In-Memory Computing (IMC) architectures have been proposed to alleviate the “memory-wall bottleneck" prevalent in von-Neumann architectures. Although recent works have proposed IMC-based SNN hardware accelerators, the following key implementation aspects have been overlooked 1) the adverse effects of crossbar non-ideality on SNN performance due to repeated analog dot-product operations over multiple time-steps 2) hardware overheads of essential SNN-specific components such as the LIF/IF and data communication modules. To this end, we propose SpikeSim, a tool that can perform realistic performance, energy, latency and area evaluation of IMC-mapped SNNs. SpikeSim consists of a practical monolithic IMC architecture called SpikeFlow for mapping SNNs. Additionally, the non-ideality computation engine (NICE) and energy-latency-area (ELA) engine performs hardware-realistic evaluation of SpikeFlow-mapped SNNs. Based on 65nm CMOS implementation and experiments on CIFAR10, CIFAR100 and TinyImagenet datasets, we find that the LIF/IF neuronal module has significant area contribution (>11% of the total hardware area). To this end, we propose SNN topological modifications that leads to 1.24× and 10× reduction in the neuronal module’s area and the overall energy-delay-product value, respectively. Furthermore, in this work, we perform a holistic comparison between IMC implemented ANN and SNNs and conclude that lower number of time-steps are the key to achieve higher throughput and energy-efficiency for SNNs compared to 4-bit ANNs. The code repository for the SpikeSim tool is available at https://github.com/Intelligent-Computing-Lab-Yale/Quanitzation-aware-SNN-training-and-hardware-evaluation-for-IMC-Architectures